Skip to content

CMG-1093 | Contentful migration got stuck some time - #1122

Merged
umesh-more-cstk merged 3 commits into
devfrom
bugfix/cmg-1093-contentful-migration-issue
Jul 28, 2026
Merged

CMG-1093 | Contentful migration got stuck some time#1122
umesh-more-cstk merged 3 commits into
devfrom
bugfix/cmg-1093-contentful-migration-issue

Conversation

@chetan-contentstack

Copy link
Copy Markdown

🔗 Jira Ticket

CMG-1093 — Delta migration | v1.0.0 | Contentful migration got stuck some time


📋 PR Type

  • ✨ Feature
  • 🐛 Bug Fix
  • 🔥 Hotfix
  • ♻️ Refactor
  • 🧹 Chore / Dependency Update
  • 📝 Documentation

📝 Description

What changed?

  • writeOneFile now uses fs.promises.writeFile and is properly awaited. It was previously calling the callback-based fs.writeFile in a fire-and-forget pattern, so the outer await writeOneFile(...) returned before the file was actually written to disk.
  • In createAssets, added fs.promises.mkdir(assetsSave, { recursive: true }) before writing the assets schema/failed files, so the destination directory is guaranteed to exist.

Why?

During delta Contentful migrations the process would occasionally hang or leave a partial state. Root cause: the assets step kicked off a non-awaited write and moved on, and if the assetsSave directory did not yet exist the callback error was swallowed via console.error, leaving downstream steps waiting on a file that was never written. Both issues are fixed here.


🧩 Affected Areas

  • api — Node.js backend
  • ui — React frontend
  • upload-api — Upload API server
  • docker / docker-compose
  • CI / GitHub Actions workflows
  • Environment variables / config
  • Other:

🧪 How to Test

  1. Run a Contentful delta migration on a project that produced the hang in v1.0.0 (any project with assets works).
  2. Watch the API logs and the assets/ output directory as the assets step runs.
  3. Confirm the migration proceeds past the assets step without stalling, and that assets/<project>/assets.json (and the failed-assets file, if any) are written before the next step begins.

Expected result: Migration completes end-to-end; the assets directory is created if missing, assets.json is fully written before subsequent steps run, and no Error writing file: 3 messages appear in logs.


📸 Screenshots / Recordings

N/A — backend-only fix, no UI changes.


🔗 Related PRs / Dependencies

  • None

✅ Author Checklist

  • Branch follows naming convention: bugfix/cmg-1093-contentful-migration-issue
  • Jira ticket linked above
  • Self-reviewed the diff — no debug logs, commented-out code, or TODOs left in
  • .env / example.env updated if new environment variables were added — N/A
  • No sensitive credentials or secrets committed
  • Existing tests pass locally (npm test)
  • New tests written (or not applicable — no unit test exists for this file path and the fix is a race-condition/IO ordering change best verified via the manual repro above)
  • README.md / docs updated if behaviour changed — N/A
  • Talisman pre-push scan passes (no secrets flagged)

👀 Reviewer Notes

Please focus on:

  • The writeOneFile change — confirm no other caller relies on the previous non-awaiting behaviour (grep shows all call sites already await it).
  • The mkdir placement in createAssets — it runs after Promise.all(tasks) and before the first writeOneFile in that block, so both the schema file and the failed-assets file land in an existing directory.

@github-actions

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check Type Count (with fixes) Without fixes Threshold Result
🔴 Critical Severity 0 0 10 ✅ Passed
🟠 High Severity 14 390 25 ✅ Passed
🟡 Medium Severity 13 12 500 ✅ Passed
🔵 Low Severity 1 0 1000 ✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

Severity Breaches (with fixes) Breaches (no fixes) SLA Threshold (with/no fixes) Status
🔴 Critical 0 0 15 / 30 days ✅ Passed
🟠 High 0 0 30 / 120 days ✅ Passed
🟡 Medium 0 0 90 / 365 days ✅ Passed
🔵 Low 0 0 180 / 365 days ✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 390
  • Medium without fixes: 12
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@chetan-contentstack chetan-contentstack changed the title fix: improve file writing and ensure asset directory creation CMG-1093 | Contentful migration got stuck some time Jul 27, 2026

@umesh-more-cstk umesh-more-cstk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMMENT review - non-gating. The human decides the merge gate.

Verdict: clean, correct bug fix. I traced the root cause and both changes and they hold up.

Verified-good

  • writeOneFile (line 338): the old callback-based fs.writeFile was fire-and-forget, so await writeOneFile(...) resolved before the bytes hit disk. migration.service.ts awaits createAssets and then immediately runs createEntry (lines 523 -> 535, and 948), and the entries step reads assets.json to resolve asset references - a write/read race that matches the reported "stuck / partial state" symptom. Switching to await fs.promises.writeFile fixes it. writeOneFile is module-private (not exported), and all three call sites - 361 inside writeFile's try/catch, and 800 & 814 inside createAssets' try/catch - already handle a rejection, so the now-throwing behavior introduces no unhandled rejection.
  • mkdir(assetsSave, { recursive: true }) (line 797): necessary and correctly placed. Lines 800 & 814 write via writeOneFile directly, which does NOT mkdir (unlike the writeFile helper at line 355 used on line 816). assetsSave is otherwise only created as a side effect of a successful per-asset write (saveAsset, line 722, on the files/<id> subdir). If every asset fails or throws before any success, assetsSave would not exist and the two direct writeOneFile writes would ENOENT - exactly what this guards.

Non-blocking

  • question - error swallowing after the fix: see inline on line 800.
  • nit - no automated test accompanies an IO-ordering fix (author acknowledged this in the checklist). Given the race nature, a test asserting assets.json exists once createAssets resolves would lock in the fix and prevent regression.

Scope: diff is a single file in api, matching the stated Affected Area and the ticket. No unrelated churn.

Comment thread api/src/services/contentful.service.ts
Comment thread api/src/services/contentful.service.ts
@github-actions

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check Type Count (with fixes) Without fixes Threshold Result
🔴 Critical Severity 0 0 10 ✅ Passed
🟠 High Severity 14 390 25 ✅ Passed
🟡 Medium Severity 13 12 500 ✅ Passed
🔵 Low Severity 1 0 1000 ✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

Severity Breaches (with fixes) Breaches (no fixes) SLA Threshold (with/no fixes) Status
🔴 Critical 0 0 15 / 30 days ✅ Passed
🟠 High 0 0 30 / 120 days ✅ Passed
🟡 Medium 0 0 90 / 365 days ✅ Passed
🔵 Low 0 0 180 / 365 days ✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 390
  • Medium without fixes: 12
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@umesh-more-cstk umesh-more-cstk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at head 86c00d7d (COMMENT — non-gating)

Re-reviewed after the new commit 86c00d7d "fix(contentful): rethrow from createAssets so write failures surface" landed on top of the previously-reviewed 0757fd0e. The only change since the last review is the single added line throw err; in the createAssets catch block. The diff is still confined to api/src/services/contentful.service.ts (3 hunks) — scope is clean and matches the api Affected Area.

Status of the three points from the prior review (0757fd0e):

  1. nit — writeOneFile now rejects instead of swallowing. Resolved by design. All three call sites (createAssets L800/L814 inside its try, and writeFile L361 inside its own try) are covered by a catch, and the new throw err is exactly what makes a writeOneFile rejection surface. Correct.
  2. question — does downstream createEntry tolerate a missing/partial assets.json after a failed write? Effectively addressed for the same run: startTestMigration/startMigration run the Contentful steps as a sequential await chain, so a throw from createAssets now aborts before createTaxonomy/createEntry/createVersionFile run — createEntry is no longer reached with a half-written assets.json. See the blocker below for the caveat on how that abort surfaces.
  3. nit — no accompanying test. Still not addressed; no test file in the diff. Non-blocking (see body note).

Blocking

  • contentful.service.ts:833 — the propagated error lands on a fire-and-forget promise with no handler. The controller invokes migrationService.startTestMigration(req) / startMigration(req) without await and without .catch() (api/src/controllers/migration.controller.ts:27 and :40), responds 200 immediately, and the migration runs detached. There is no global process.on('unhandledRejection') anywhere in api and no --unhandled-rejections flag in the start scripts. So the new throw err turns an asset write/mkdir/read failure into an unhandled promise rejection, which on Node's default (throw, v15+) terminates the whole API process — taking down every in-flight migration — while the client already got a 200 and never learns it failed. Before this PR, createAssets swallowed the error and continued. Details/fix options inline.

Non-blocking

  • No test (prior nit #3, still open). The author's checklist rationale (race-condition/IO fix) is now weaker: with the await + throw, the behavior is deterministic and unit-testable (e.g. stub fs.promises.writeFile to reject and assert createAssets rejects; stub it to resolve and assert assets.json content). A small test would lock in the fix.

Verified good

  • writeOneFile await fs.promises.writeFile (L338) — the previous callback form genuinely resolved the outer await before the write completed; this is a real fix.
  • mkdir(assetsSave, { recursive: true }) (L797) — necessary and correct: when every asset fails, saveAsset never creates a dir under assetsSave (it only mkdirs assetPath on the success path), so the subsequent writeOneFile(assets.json) would ENOENT without this. recursive makes it idempotent for the normal path.
  • The failure is still logged via customLogger immediately before the throw, so it is not silent in the logs — the gap is purely the missing top-level handler.

Out-of-diff notes (can't anchor inline)

  • The fire-and-forget controller (migration.controller.ts:27/:40) and the absence of a global rejection handler are pre-existing and systemic — aem.service.ts createAssets already propagates (no outer try/catch), so this PR brings Contentful in line with that pattern rather than inventing it. That's why the recommended fix likely belongs in startTestMigration/startMigration (wrap the switch, mark the project failed, log) or in the controller, not necessarily in this file.

Comment thread api/src/services/contentful.service.ts
Comment thread api/src/services/contentful.service.ts
@github-actions

Copy link
Copy Markdown

🔒 Security Scan Results

ℹ️ Note: Only vulnerabilities with available fixes (upgrades or patches) are counted toward thresholds.

Check Type Count (with fixes) Without fixes Threshold Result
🔴 Critical Severity 0 0 10 ✅ Passed
🟠 High Severity 14 390 25 ✅ Passed
🟡 Medium Severity 13 12 500 ✅ Passed
🔵 Low Severity 1 0 1000 ✅ Passed

⏱️ SLA Breach Summary

✅ No SLA breaches detected. All vulnerabilities are within acceptable time thresholds.

Severity Breaches (with fixes) Breaches (no fixes) SLA Threshold (with/no fixes) Status
🔴 Critical 0 0 15 / 30 days ✅ Passed
🟠 High 0 0 30 / 120 days ✅ Passed
🟡 Medium 0 0 90 / 365 days ✅ Passed
🔵 Low 0 0 180 / 365 days ✅ Passed

ℹ️ Vulnerabilities Without Available Fixes (Informational Only)

The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:

  • Critical without fixes: 0
  • High without fixes: 390
  • Medium without fixes: 12
  • Low without fixes: 0

✅ BUILD PASSED - All security checks passed

@umesh-more-cstk
umesh-more-cstk merged commit b53a2e8 into dev Jul 28, 2026
9 checks passed
@umesh-more-cstk
umesh-more-cstk deleted the bugfix/cmg-1093-contentful-migration-issue branch July 28, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants